{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "jupyter": {
     "outputs_hidden": false
    }
   },
   "source": [
    "## 1. PP-ShiTu模型简介\n",
    "PP-ShiTu是一个实用的轻量级通用图像识别系统，主要由主体检测、特征学习和向量检索三个模块组成。该系统从骨干网络选择和调整、损失函数的选择、数据增强、学习率变换策略、正则化参数选择、预训练模型使用以及模型裁剪量化8个方面，采用多种策略，对各个模块的模型进行优化，最终得到在CPU上仅0.2s即可完成10w+库的图像识别的系统。\n",
    "更多细节请参考 [PP-ShiTu技术方案](https://arxiv.org/pdf/2111.00775.pdf)。\n",
    "\n",
    "更多关于PaddleClas可以点击 https://github.com/PaddlePaddle/PaddleClas 进行了解。\n",
    "\n",
    "## 2. 模型效果及应用场景\n",
    "### 2.1 商品识别任务：\n",
    "\n",
    "#### 2.1.1 数据集：\n",
    "\n",
    "PP-ShiTu的训练数据集和测试集由 Aliproduct、GLDv2等共7个数据集组成，详细信息可参考 [PP-ShiTu 实验部分](https://github.com/PaddlePaddle/PaddleClas/blob/release/2.4/docs/zh_CN/image_recognition_pipeline/feature_extraction.md#4-%E5%AE%9E%E9%AA%8C%E9%83%A8%E5%88%86)\n",
    "\n",
    "#### 2.1.2 模型效果速览：\n",
    "\n",
    "PP-ShiTu 在图片上的检测效果如下\n",
    "\n",
    "![](https://github.com/PaddlePaddle/PaddleClas/raw/release/2.4/docs/images/recognition/drink_data_demo/output/nongfu_spring.jpeg)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 3. 模型如何使用\n",
    "\n",
    "### 3.1 模型推理：\n",
    "\n",
    "- 下载 PaddleClas"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true,
    "execution": {
     "iopub.execute_input": "2022-11-08T08:24:16.514016Z",
     "iopub.status.busy": "2022-11-08T08:24:16.513368Z",
     "iopub.status.idle": "2022-11-08T08:25:00.630629Z",
     "shell.execute_reply": "2022-11-08T08:25:00.629113Z",
     "shell.execute_reply.started": "2022-11-08T08:24:16.513971Z"
    },
    "jupyter": {
     "outputs_hidden": true
    },
    "scrolled": true,
    "tags": []
   },
   "outputs": [],
   "source": [
    "# 不在Jupyter Notebook上运行时需要将含 \"!\" 和 \"%\" 的语句注释，不需要运行。\n",
    "%cd ~/work\n",
    "\n",
    "# 克隆 PaddleClas（gitee上克隆速度较快）\n",
    "!git clone https://gitee.com/paddlepaddle/PaddleClas"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- 安装 PaddleClas 及其依赖包"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true,
    "execution": {
     "iopub.execute_input": "2022-11-08T08:26:02.622321Z",
     "iopub.status.busy": "2022-11-08T08:26:02.621656Z",
     "iopub.status.idle": "2022-11-08T08:26:05.016413Z",
     "shell.execute_reply": "2022-11-08T08:26:05.015052Z",
     "shell.execute_reply.started": "2022-11-08T08:26:02.622277Z"
    },
    "jupyter": {
     "outputs_hidden": true
    },
    "scrolled": true,
    "tags": []
   },
   "outputs": [],
   "source": [
    "# 进入 PaddleClas 目录\n",
    "%cd ~/work/PaddleClas/\n",
    "\n",
    "# 切换到2.4分支\n",
    "!git checkout release/2.4\n",
    "\n",
    "# 安装所需依赖项\n",
    "!pip install -r requirements.txt\n",
    "\n",
    "# 设置GPU\n",
    "# %env CUDA_VISIBLE_DEVICES=0"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- 快速体验\n",
    "\n",
    "恭喜！ 您已经成功安装了 PaddleClas，接下来快速体验图像识别效果"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2022-11-08T08:26:11.091828Z",
     "iopub.status.busy": "2022-11-08T08:26:11.090376Z",
     "iopub.status.idle": "2022-11-08T08:29:06.202735Z",
     "shell.execute_reply": "2022-11-08T08:29:06.201197Z",
     "shell.execute_reply.started": "2022-11-08T08:26:11.091754Z"
    },
    "scrolled": true,
    "tags": []
   },
   "outputs": [],
   "source": [
    "# 进入 PaddleClas 目录\n",
    "%cd ~/work/PaddleClas/\n",
    "\n",
    "# 创建存放主体检测、特征提取推理模型的文件夹\n",
    "%mkdir -p deploy/models\n",
    "\n",
    "# 进入该文件夹\n",
    "%cd deploy/models\n",
    "\n",
    "# 下载主体检测inference模型并解压\n",
    "!wget -nc https://paddle-imagenet-models-name.bj.bcebos.com/dygraph/rec/models/inference/picodet_PPLCNet_x2_5_mainbody_lite_v1.0_infer.tar && tar -xf picodet_PPLCNet_x2_5_mainbody_lite_v1.0_infer.tar\n",
    "\n",
    "# 下载特征提取inference模型并解压\n",
    "!wget -nc https://paddle-imagenet-models-name.bj.bcebos.com/dygraph/rec/models/inference/general_PPLCNet_x2_5_lite_v1.0_infer.tar && tar -xf general_PPLCNet_x2_5_lite_v1.0_infer.tar\n",
    "\n",
    "# 返回至deploy文件夹\n",
    "%cd ~/work/PaddleClas/deploy/\n",
    "\n",
    "# 下载测试数据 drink_dataset_v1.0 并解压\n",
    "!wget -nc https://paddle-imagenet-models-name.bj.bcebos.com/dygraph/rec/data/drink_dataset_v1.0.tar && tar -xf drink_dataset_v1.0.tar"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2022-11-08T08:31:30.364422Z",
     "iopub.status.busy": "2022-11-08T08:31:30.363351Z",
     "iopub.status.idle": "2022-11-08T08:31:37.682006Z",
     "shell.execute_reply": "2022-11-08T08:31:37.680563Z",
     "shell.execute_reply.started": "2022-11-08T08:31:30.364378Z"
    },
    "scrolled": true,
    "tags": []
   },
   "outputs": [],
   "source": [
    "# 进入 PaddleClas 目录\n",
    "%cd ~/work/PaddleClas/\n",
    "\n",
    "# 进入deploy文件夹\n",
    "%cd ./deploy\n",
    "\n",
    "# 用 general_PPLCNet_x2_5_lite_v1.0 推理模型提取gallery图片的特征，制作成检索库\n",
    "!python3.7 python/build_gallery.py -c configs/inference_general.yaml -o Global.rec_inference_model_dir=./models/general_PPLCNet_x2_5_lite_v1.0_infer\n",
    "\n",
    "# 对 nongfu_spring.jpeg 图片进行识别推理(GPU推理)\n",
    "!python3.7 python/predict_system.py -c configs/inference_general.yaml\n",
    "# 对 nongfu_spring.jpeg 图片进行识别推理(CPU推理)\n",
    "!python3.7 python/predict_system.py -c configs/inference_general.yaml -o Global.use_gpu=False"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "同时识别的结果（带有检测框、对应类别以及相似度）会保存至 `PaddleClas/deploy/output/nongfu_spring.jpeg`\n",
    "\n",
    "![](https://github.com/PaddlePaddle/PaddleClas/raw/release/2.4/docs/images/recognition/drink_data_demo/output/nongfu_spring.jpeg)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3.2 模型训练\n",
    "\n",
    "- 克隆 PaddleClas 仓库（参考 3.1模型推理 - 下载PaddleClas），并切换到2.4分支\n",
    "- 主体检测模型的数据集准备、开始训练、模型评估等步骤，请参考 [PP-ShiTu 主体检测 文档](https://github.com/PaddlePaddle/PaddleClas/blob/release/2.4/docs/zh_CN/image_recognition_pipeline/mainbody_detection.md)\n",
    "- 特征提取模型的数据集准备、开始训练、模型评估等步骤，请参考 [PP-ShiTu 特征提取 文档](https://github.com/PaddlePaddle/PaddleClas/blob/release/2.4/docs/zh_CN/image_recognition_pipeline/feature_extraction.md#51-%E6%95%B0%E6%8D%AE%E5%87%86%E5%A4%87)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 4. 模型原理\n",
    "PP-ShiTu 系列识别系统，包括本文档介绍的 PP-ShiTu，均由3个模块串联完成整个识别过程，如下图所示\n",
    "\n",
    "![PP-ShiTu系统](https://github.com/PaddlePaddle/PaddleClas/blob/release/2.4/docs/images/structure.jpg?raw=true)\n",
    "\n",
    "- 主体检测：上图中的蓝色模块，主要负责检测出用户输入图片中可能的识别目标，进而裁剪出这些目标，过滤不重要的背景，减少背景的干扰。事实上这种保留主体，过滤背景的做法是实践中会采用的一种简单而有效的方法。\n",
    "- 特征提取：接收 **主体检测** 模块输出的含有目标主体的裁剪后的图片，将其输入到特征提取模型中，得到对应的特征向量，作为该图片的表示特征用于接下来的检索步骤。\n",
    "- 向量检索：接收 **特征提取** 模块输出的一个或多个特征向量，逐个地在向量库中检索，将检索库中最邻近（一般以相似度表示邻近程度）的向量的类别，作为检索向量的类别，最后返回检索结果。该模块不需要额外训练，安装第三方开源的faiss检索库即可使用\n",
    "\n",
    "在检索系统中，最重要的模块之一就是特征提取模型，其特征提取能力好坏直接影响检索库内向量和待检索向量的质量，因此接下来分5个部分，重点介绍 PP-ShiTu 所使用的特征提取模型。\n",
    "\n",
    "- Backbone\n",
    "    Backbone 部分采用了 PP_LCNet_x2_5，其针对Intel CPU端的性能优化探索了多个有效的结构设计方案，最终实现了在不增加推理时间的情况下，进一步提升模型的性能，最终大幅度超越现有的 SOTA 模型。\n",
    "\n",
    "- Neck\n",
    "\n",
    "    Neck 部分采用了 FC Layer，对 Backbone 抽取得到的特征进行降维，减少了特征存储的成本与计算量。\n",
    "\n",
    "- Head\n",
    "\n",
    "    Head 部分选用 ArcMargin，在训练时通过指定margin，增大同类特征之间的角度差异再进行分类，进一步提升抽取特征的表征能力。\n",
    "\n",
    "- Loss\n",
    "\n",
    "    Loss 部分选用 Cross entropy loss，在训练时以分类任务的损失函数来指导网络进行优化。详细的配置文件见通用识别配置文件。\n",
    "\n",
    "## 5. 注意事项\n",
    "PP-ShiTu 是在寻找在产业实践中最高性价比的图像识别方案，但考虑到不同识别场景的数据集均有各自的分布特点，以及训练时的软硬件限制，无法一次性将所有的数据集全部纳入到训练集中，经过权衡才使用了目前这套训练数据集的组合。因此推荐用户在了解自己实际业务数据集的特点之后，基于 PP-ShiTu 的预训练模型以及训练配置，在自己的业务数据集上进行微调甚至二次开发，以获得性能更好，更适配自己数据集的识别模型。\n",
    "\n",
    "## 6. 相关论文以及引用信息\n",
    "```log\n",
    "@article{cui2021pp,\n",
    "    title={PP-LCNet: A Lightweight CPU Convolutional Neural Network},\n",
    "    author={Cui, Cheng and Gao, Tingquan and Wei, Shengyu and Du, Yuning and Guo, Ruoyu and Dong, Shuilong and Lu, Bin and Zhou, Ying and Lv, Xueying and Liu, Qiwen and others},\n",
    "    journal={arXiv preprint arXiv:2109.15099},\n",
    "    year={2021}\n",
    "}\n",
    "\n",
    "@article{wei2021pp,\n",
    "    title={PP-ShiTu: A Practical Lightweight Image Recognition System},\n",
    "    author={Wei, Shengyu and Guo, Ruoyu and Cui, Cheng and Lu, Bin and Dong, Shuilong and Gao, Tingquan and Du, Yuning and Zhou, Ying and Lyu, Xueying and Liu, Qiwen and others},\n",
    "    journal={arXiv preprint arXiv:2111.00775},\n",
    "    year={2021}\n",
    "}\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "请点击[此处](https://ai.baidu.com/docs#/AIStudio_Project_Notebook/a38e5576)查看本环境基本用法.  <br>\n",
    "Please click [here ](https://ai.baidu.com/docs#/AIStudio_Project_Notebook/a38e5576) for more detailed instructions. "
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "py35-paddle1.2.0"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}
